| iv | dv |
|---|---|
| HIGH | No |
| HIGH | No |
| LOW | No |
| HIGH | Yes |
| HIGH | Yes |
| HIGH | Yes |
| HIGH | Yes |
| LOW | Yes |
| LOW | Yes |
| LOW | Yes |
Chapter 4: Research Design
Cross tabulation: interpretation
Look at what happens to the DV at different values of the IV. If your variables are ordinal, you should be able to identify a direction of the effect.
The proportion of “Yes” values decreases as the IV goes from lower to higher, so this is a negative or inverse relationship.
iv
|
||
|---|---|---|
| dv | LOW | HIGH |
| No | 1 (25%) | 2 (33%) |
| Yes | 3 (75%) | 4 (67%) |
| Total | 4 | 6 |
Using a bar graph or line graph can make these relationships easier to spot:
Cross tabulation: example
Are these results generally consistent with my hypothesis?
Party ID
|
|||
|---|---|---|---|
| turnout2020 | Democrat | Independent | Republican |
| 0. Did not vote | 335 (10%) | 316 (36%) | 382 (12%) |
| 1. Voted | 3160 (90%) | 560 (64%) | 2714 (88%) |
If we think of party ID as an ordered variable, this is a curvilinear relationship.
Mean Comparison
Here again, the relationship can be easier to conceptualize if we plot it.
Curvilinear Relationships
A relationship like this will rarely be perfectly straight, so “linearity” and “curvilinearity” are partly a matter of degree, but there are some cases where there is a clear “U” shape to the relationship:
| iv | dv |
|---|---|
| 1. Extremely liberal | 6.314 |
| 2. Liberal | 5.685 |
| 3. Slightly liberal | 5.001 |
| 4. Moderate; middle of the road | 4.651 |
| 5. Slightly conservative | 4.636 |
| 6. Conservative | 4.974 |
| 7. Extremely conservative | 5.363 |
Experiments: Field Experiments
Field experiments can lessen the external validity problem by using random assignment in the field.
For instance, one common way to study GOTV messaging is to randomly select households to receive mailers:
Civic duty treatment
Hawthorne treatment
Neighbors treatment 
Self treatment
From: GERBER, A. S., GREEN, D. P., & LARIMER, C. W. (2008). Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment. American Political Science Review, 102(1), 33–48. doi:10.1017/S000305540808009X
Experiments: Field Experiments
From: GERBER, A. S., GREEN, D. P., & LARIMER, C. W. (2008). Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment. American Political Science Review, 102(1), 33–48. doi:10.1017/S000305540808009X
Experiments: Natural Experiments
Viewing Fox News isn’t random, but areas where Fox News is lower in the channel order will have more viewers.
Channel order is essentially randomly assigned.
So, using channel order as a “treatment” assignment might theoretically allow us to account for confounding in an observational setting.
The 1936 Literary Digest Poll
Literary Digest wanted to predict the outcome of the 1936 presidential election.
Mailed out approx. 10 million “ballots” based on address data from social clubs, automobile registrations, phone books etc. about 2.4 million were returned.
10 million much larger than the target for most polls
A 24% response rate was low for the time, but higher than contemporary polls
Scientific polling was in its infancy, Literary Digest touted their lack of any fancy data manipulation as an advantage over polls that used more complex methodologies.
The 1936 Literary Digest poll
| Prediction for Landon | Prediction for Roosevelt |
| 54% of the popular vote | 41% of the popular vote |
The 1936 Literary Digest poll
Not quite!
The 1936 Literary Digest poll
Contemporaneous analyses attributed the error to sampling bias: Literary Digest targeted car owners and telephone owners, and so the results skewed toward the wealthy
Subsequent reanalyses point to non-response bias: Roosevelt supporters were systematically less likely to return the postcard compared to Landon supporters.
Anecdotal evidence suggests that this may have been the result of different levels of enthusiasm: Landon voters really didn’t like Roosevelt and they were motivated to talk about it.
Sample size doesn’t fix bias! Much smaller random polls can easily outperform a large-yet-biased one.
Sampling strategies
A note that’s still relevant today: non-response bias is a big risk! Some people are more likely to take polls.
Sampling strategies
Contemporary polls often get stuff wrong, but many general election pollsters are able to get a lot closer with far fewer observations (even as conditions get worse!) How?
Sampling strategies: Simple Random Samples
Simple Random Samples: get a list of the entire target population, and start selecting at random.
Pros: we’ll converge toward representativeness +/- some random error. As we approach sample sizes of 1,000 or more, the probability of a very large random error becomes negligible.
Cons:
Where am I supposed to get a list of the entire population!?
Can be inefficient: we need a lot of data to get the margin of error to an acceptable level
What if I want to study public opinion among a small group? Say: convicted felons, LGBTQ people, or millionaires?
Sampling strategies: Stratification and Clustering
Stratified Sampling divides the population into groups and then randomly samples within those groups.
- For instance: I might group people by generation (Z, millennial, gen-X, etc.) and then randomly sample until I hit a target quota for generation.
Pros: Ensuring representativeness isn’t left entirely to chance, and I can even oversample certain groups if they’re hard to reach or rare in the population.
Cons: this isn’t pure random sampling, so we will need to re-weight the data to make it look like the actual population and account for this when calculating our margin of error.
Sampling strategies: Stratification and Clustering
Cluster Sampling: Instead of targeting individuals, I might target a geographic area like a household, or a city block, or a census tract.
Pros: Potentially much easier to target a random sample of something like households.
Cons: again, not a random sample, so we may need to do some re-weighting and account for this when calculating our margin of error.
Observational data and control
- Assuming we’ve collected good observational data: how do we control for confounding without randomization?
- We’ll address this issue using controlled comparisons: essentially, we will split the data up by values of a potential confounder, and see whether the same general relationship still shows up among groups with the same values on the confounding variable.
